Foundations, Inference, and Deconvolution in Image Restoration

Image restoration is a critical preprocessing step in computer vision, producing images with reduced noise, blur, and pixel defects. This enables precise higher-level reasoning as to the scene content in later stages of the vision pipeline (e.g., object segmentation, detection, recognition, and tracking). Restoration techniques have found extensive usage in a broad range of applications from industry, medicine, astronomy, biology, and photography. The recovery of high-grade results requires models of the image degradation process, giving rise to a class of often heavily underconstrained, inverse problems. A further challenge specific to the problem of blur removal is noise amplification, which may cause strong distortion by ringing artifacts. This dissertation presents new insights and problem solving procedures for three areas of image restoration, namely (1) model foundations, (2) Bayesian inference for high-order Markov random fields (MRFs), and (3) blind image deblurring (deconvolution). As basic research on model foundations, we contribute to reconciling the perceived differences between probabilistic MRFs on the one hand, and deterministic variational models on the other. To do so, we restrict the variational functional to locally supported finite elements (FE) and integrate over the domain. This yields a sum of terms depending locally on FE basis coefficients, and by identifying the latter with pixels, the terms resolve to MRF potential functions. In contrast with previous literature, we place special emphasis on robust regularizers used commonly in contemporary computer vision. Moreover, we draw samples from the derived models to further demonstrate the probabilistic connection. Another focal issue is a class of high-order Field of Experts MRFs which are learned generatively from natural image data and yield best quantitative results under Bayesian estimation. This involves minimizing an integral expression, which has no closed form solution in general. However, the MRF class under study has Gaussian mixture potentials, permitting expansion by indicator variables as a technical measure. As approximate inference method, we study Gibbs sampling in the context of non-blind deblurring and obtain excellent results, yet at the cost of high computing effort. In reaction to this, we turn to the mean field algorithm, and show that it scales quadratically in the clique size for a standard restoration setting with linear degradation model. An empirical study of mean field over several restoration scenarios confirms advantageous properties with regard to both image quality and computational runtime. This dissertation further examines the problem of blind deconvolution, beginning with localized blur from fast moving objects in the scene, or from camera defocus. Forgoing dedicated hardware or user labels, we rely only on the image as input and introduce a latent variable model to explain the non-uniform blur. The inference procedure estimates freely varying kernels and we demonstrate its generality by extensive experiments. We further present a discriminative method for blind removal of camera shake. In particular, we interleave discriminative non-blind deconvolution steps with kernel estimation and leverage the error cancellation effects of the Regression Tree Field model to attain a deblurring process with tightly linked sequential stages

Foundations, Inference, and Deconvolution in Image Restoration

Image restoration is a critical preprocessing step in computer vision, producing images with reduced noise, blur, and pixel defects. This enables precise higher-level reasoning as to the scene content in later stages of the vision pipeline (e.g., object segmentation, detection, recognition, and tracking). Restoration techniques have found extensive usage in a broad range of applications from industry, medicine, astronomy, biology, and photography. The recovery of high-grade results requires models of the image degradation process, giving rise to a class of often heavily underconstrained, inverse problems. A further challenge specific to the problem of blur removal is noise amplification, which may cause strong distortion by ringing artifacts. This dissertation presents new insights and problem solving procedures for three areas of image restoration, namely (1) model foundations, (2) Bayesian inference for high-order Markov random fields (MRFs), and (3) blind image deblurring (deconvolution). As basic research on model foundations, we contribute to reconciling the perceived differences between probabilistic MRFs on the one hand, and deterministic variational models on the other. To do so, we restrict the variational functional to locally supported finite elements (FE) and integrate over the domain. This yields a sum of terms depending locally on FE basis coefficients, and by identifying the latter with pixels, the terms resolve to MRF potential functions. In contrast with previous literature, we place special emphasis on robust regularizers used commonly in contemporary computer vision. Moreover, we draw samples from the derived models to further demonstrate the probabilistic connection. Another focal issue is a class of high-order Field of Experts MRFs which are learned generatively from natural image data and yield best quantitative results under Bayesian estimation. This involves minimizing an integral expression, which has no closed form solution in general. However, the MRF class under study has Gaussian mixture potentials, permitting expansion by indicator variables as a technical measure. As approximate inference method, we study Gibbs sampling in the context of non-blind deblurring and obtain excellent results, yet at the cost of high computing effort. In reaction to this, we turn to the mean field algorithm, and show that it scales quadratically in the clique size for a standard restoration setting with linear degradation model. An empirical study of mean field over several restoration scenarios confirms advantageous properties with regard to both image quality and computational runtime. This dissertation further examines the problem of blind deconvolution, beginning with localized blur from fast moving objects in the scene, or from camera defocus. Forgoing dedicated hardware or user labels, we rely only on the image as input and introduce a latent variable model to explain the non-uniform blur. The inference procedure estimates freely varying kernels and we demonstrate its generality by extensive experiments. We further present a discriminative method for blind removal of camera shake. In particular, we interleave discriminative non-blind deconvolution steps with kernel estimation and leverage the error cancellation effects of the Regression Tree Field model to attain a deblurring process with tightly linked sequential stages

Mean Field for Continuous High-Order MRFs

Abstract. Probabilistic inference beyond MAP estimation is of interest in computer vision, both for learning appropriate models and in applications. Yet, common approximate inference techniques, such as belief propagation, have largely been limited to discrete-valued Markov random fields (MRFs) and models with small cliques. Oftentimes, neither is desirable from an application standpoint. This paper studies mean field inference for continuous-valued MRF models with high-order cliques. Mean field can be applied effectively to such models by exploiting that the factors of certain classes of MRFs can be formulated using Gaussian mixtures, which allows retaining the mixture indicator as a latent variable. We use an image restoration setting to show that resulting mean field updates have a computational complexity quadratic in the clique size, which makes them scale even to large cliques. We contribute an empirical study with four applications: Image denoising, non-blind deblurring, noise estimation, and layer separation from a single image. We find mean field to yield a favorable combination of performance and efficiency, e.g. outperforming MAP estimation in denoising while being competitive with expensive sampling approaches. Novel approaches to noise estimation and layer separation demonstrate the breadth of applicability.

Connecting non-quadratic variational models and MRFs

Spatially-discrete Markov random fields (MRFs) and spatially-continuous variational approaches are ubiquitous in low-level vision, including image restoration, segmentation, optical flow, and stereo. Even though both families of approaches are fairly similar on an intuitive level, they are frequently seen as being technically rather distinct since they operate on different domains. In this paper we explore their connections and develop a direct, rigorous link with a particular emphasis on first-order regularizers. By representing spatially-continuous functions as linear combinations of finite elements with local support and performing explicit integration of the variational objective, we derive MRF potentials that make the resulting MRF energy equivalent to the variational energy functional. In contrast to previous attempts, we provide an explicit connection for modern non-quadratic regularizers and also integrate the data term. The established connection opens certain classes of MRFs to spatially-continuous interpretations and variational formulations to a broad range of probabilistic learning and inference algorithms

Bayesian deblurring with integrated noise estimation

Conventional non-blind image deblurring algorithms involve natural image priors and maximum a-posteriori (MAP) estimation. As a consequence of MAP estimation, separate pre-processing steps such as noise estimation and training of the regularization parameter are necessary to avoid user interaction. Moreover, MAP estimates involving standard natural image priors have been found lacking in terms of restoration performance. To address these issues we introduce an integrated Bayesian framework that unifies non-blind deblurring and noise estimation, thus freeing the user of tediously pre-determining a noise level. A sampling based technique allows to integrate out the unknown noise level and to perform deblurring using the Bayesian minimum mean squared error estimate (MMSE), which requires no regularization parameter and yields higher performance than MAP estimates when combined with a learned high order image prior. A quantitative evaluation demonstrates state-of-the-art results for both non-blind deblurring and noise estimation

Interleaved Regression Tree Field Cascades for Blind Image Deconvolution

Image blur from camera shake is a common cause for poor image quality in digital photography, prompting a sig-nificant recent interest in image deblurring. The vast ma-jority of work on blind deblurring splits the problem into two subsequent steps: First, the blur process (i.e., blur ker-nel) is estimated; then the image is restored given the esti-mated kernel using a non-blind deblurring algorithm. Re-cent work in non-blind deblurring has shown that discrim-inative approaches can have clear image quality and run-time benefits over typical generative formulations. In this paper, we propose a cascade for blind deblurring that alter-nates between kernel estimation and discriminative deblur-ring using regression tree fields (RTFs). We further con-tribute a new dataset of realistic image blur kernels from human camera shake, which we use to train the discrim-inative component. Extensive qualitative and quantitative experiments show a clear gain in image quality by inter-leaving kernel estimation and discriminative deblurring in an iterative cascade. 1

Supervised by

I would like to thank the following people. • Prof. Dr. Johannes Buchmann, for a fascinating thesis. • Dipl.-Math. Richard Lindner and Dipl.-Inf. Markus Rückert, for inspiring scientific guidance. Warranty I hereby warrant that the content of this thesis is the direct result of my own work and that any use made in it of published or unpublished material is fully and correctly referenced